For safety reasons, the systems designed to be placed onboard an aircraft have to undergo tests to make sure they run smoothly and show that said systems meet the certification requirements before an aircraft equipped with such systems is authorized to fly and to enter into commercial service.
Before they are installed, these systems now undergo many tests to make sure they meet the integrity, safety and other requirements issued by the certification authorities. These onboard systems can be specialized computers designed to perform functions that may be crucial for the aircraft, for example pilot functions. Hereinafter, these systems will be called computers.
Most often, in the architectures of the current systems, each computer is dedicated to one or more applications of the same kind, for example flight control applications. Each computer has a hardware part and a software part. The hardware part includes at least one central processing unit (CPU) and at least one input/output unit by which the computer is connected to a network of computers, external peripherals, etc.
One basic feature of onboard systems, often used in the field of aeronautics, is associated with an architecture, both hardware and software, that prevents the introduction, as much as possible, of any means not necessary to perform the functions dedicated to said systems.
Thus, contrary to the systems generally encountered in applications commonly used in aeronautics, the computer is not equipped with a complex operating system. Moreover, the software is written in a language as close as possible to the language understood by the central processing unit, and the only inputs/outputs available are those necessary to operate the system, for example information coming from sensors or other elements of the aircraft or information intended for actuators or other elements.
The advantage of this type of architecture is the fact that the operation of such a system is much better controlled. It is not dependent on a complex operating system, some aspects of whose operation depend on parameters that are not controlled and that would otherwise have to be subject to the same demonstrations of safety as the applications software. The system is simple and less vulnerable, since it has only the means strictly needed to perform the functions assigned to said system.
On the other hand, the operation of such a system is much harder to observe. For example, the system has no conventional man/machine interfaces, like keyboards or screens, which would make it possible to be sure the sequences of instructions are being carried out properly and to interact with them, which makes it hard to run the tests essential during software development, verification and qualification.
The software part of the computer includes software specific to the application considered and ensures the operation of the computer, whose logic instructions correspond to algorithms that determine the operation of the system.
To obtain certification for the system, prior to its commissioning and that of the aircraft, a computer validation phase is carried out.
As is known, the validation phase generally consists of verifying, at each step in the process of making the computer, that it meets the specifications that have been set so that said computer gives the operating responses expected from the system.
This conformity to specifications is done, particularly for the software, in successive steps from verifying the simplest software components to the complete software program containing all components that must be built into the target computer.
In the first verification step, the simplest software elements that can be tested are put through so-called unit tests. These tests make sure that the logic instructions, i.e., the code, for said software elements, taken individually, have been written in accordance with the design requirements.
In the second, so-called integration step, different software components that have been put through isolated tests individually are integrated to create an assembly in which the software components interact. These different software components are put through integration tests designed to make sure that the software components are compatible, particularly in terms of the functional interfaces between said components.
In the third step, all of the software components are built into the computer for which they are intended. Validation tests are then done to show that the software, made up of all the components built into the computer, meets the specification, i.e., that it performs the functions expected and that it operates reliably and safely.
To guarantee that a software program is safe and to meet the certification requirements, it is also necessary, in this validation phase, to show that all of the tests which the software has gone through make it possible to conclude, with an adequate level of assurance, that the software meets the security requirements of the system into which it is built.
The different tests done on the software during the validation phase make it possible to be sure that no malfunction of said software (which could have an impact on the proper operation of the computers and consequently on the aircraft and its safety) can occur or that, if a malfunction does occur, the software can handle it.
However, during the validation phase, and especially for the operations to investigate when there are problems, it is often necessary to make sure not only that the input and output parameters of the computer in which the software is installed meet the expectations, but also that certain internal behaviors of the software are correct.
In this case, because of the specific architecture of the specialized computers for onboard applications, it is generally very difficult to observe the operation of the software without using special devices and methods.
The first known method consists of setting up a file distribution system between the computer being tested with the software installed and an associated platform, using simulators. A simulator is understood to be a device that makes it possible to simulate, on the associated platform, the logic operation of a calculating unit, a processor of the computer.
In such an operating mode with a simulator, the computer processor is replaced by a probe that serves as an interface with the associated platform for the simulation of the processor.
Thus, it is possible to have the software being tested run on the computer, except for the processor part and, through functions proper to the associated platform, to observe the operation or certain internal malfunctions of the software, for example, in response to simulations of inputs of the input/output units, and to observe the outputs of said input/output units.
This first method has many drawbacks. Actually, each type of computer being tested requires a specific test bench or at least a very specialized configuration of a test bench. A test bench is an assembly that includes, in particular, means of interconnection with the computer being tested, means of simulating the computer processor or processors and means of running test programs.
Since each processor requires a specific simulator, both for the simulation software and for the probe connected in place of the processor, it is necessary to have multiple simulators according to the definitions of the computers.
Moreover, the possibilities of investigating with simulators are generally limited. The need to work with the specific machine language of the processor under consideration implies that the user be a specialist in machine-language programming.
What is more, a simulator is an expensive product that is generally produced only in small quantities. The life cycle of this type of product is very short (6 months to 2 years), while the means of development and verification must be maintained in operating condition (regulations, industrial reactivity) for the duration of the aircraft program (20 years or more). This translates into problems dealing with obsolescence that are increasingly difficult to solve.
This simulator solution therefore proves poorly suited, since besides its limited performance in terms of investigation, it is expensive to set up and expensive to maintain.
The cost is also hurt by the fact that different processor models are generally used to provide functional redundancy, through design safety, thus multiplying the need for simulators.
A second method, which is aimed at eliminating the simulator problems, consists of simulating the operation of the computer that must run the program being tested on a host platform. In this case, the software being tested must access files on the host platform, either to read the test vectors or to record test results.
Since the software being tested does not naturally have functions for such access to the files on the host platform, it is necessary to modify the software being tested to integrate these access functions.
To transfer the information, generally system calling instructions are used that are issued by the simulated test environment. The system calling instructions can be, for example, to open a file, write a file or read a file. The system calling instructions are intercepted by the operating system of the host platform, which converts them into system calls from the host platform.
This second method also has drawbacks. Actually, the variety of files is such that the development of access functionalities is very dependent on the host platform and its operating system. Now, the variability of the host platforms is huge both in space (for development teams scattered all over the world) and in time (replacing host platforms), which poses practical problems using the method.
These problems are accentuated by the fact that the skills of experts, who can modify the functions of the operating systems, are required to develop such functionalities to access the file systems, and therefore cannot be assigned to test specialists.
Consequently, this method has proven expensive and hard to use.
What is more, this method is very intrusive in terms of the software being tested, and modifying software to run tests on it causes the risk of interfering with the operation of the software itself.